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ABSTRACT 

Work on item response theory was extended to two 
areas not extensively researched previously, including models for: 
(1) test items that require more than one ability for a correct 
response (MIRT); and (2) interaction between modules of instruction 
that have a hierarchical relationship (HST). In order to develop the 
MIRT and HST models, the author evaluated characteristics of the 
models, developed estimation procedures for the parameter of the 
models, and evr.luated the models on their ability to describe real 
test data. A summary of the research is presented here, and 
references are made to papers and technical reports containing more 
detailed descriptions o^ the research efforts. (Author/LMO) 
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Final Report 
Models for Multidimensional Tests 
and Hierarchically Structural Training Materials 

Since the I950*s, there has been increasing interest in psychological and 
educational measurement tl^at is based upon probalistic models of the 
interaction between a person and a test item« -These model-based procedures 
demonstrate how strong assumptions can be used to gain increased control over 
the measurement proc^sT. For example, using item response theory (IRT), the 
precision of measurement at every point along an ability scale can be 
determined • Also, items can be selected from a pool to foinn a test with any 
desired level of precision at any point on the score scale* 

The strong assumptions needed for these model-based procedures are 
basically that the probabilistic model that has been selected accurately 
reflects the test data, and that local independence holds for the model* This 
latter assumption means that the response to one item does not affect the 
response to another item, and that the response by one person does not affect 
the response by another person. 

Most of the current models assume that the measuring instrument measures 
only a single trait (Rasch, 1960; Lord, 1952; Birnbaum, 1968). For many 
tests, this assumption is at least approximated, and for other tests, it is 
unlikely to be met at all. Most of the current models also are limited to 
describing a person's response to a single item. In some cases this 
limitation may make it difficult to solve some measurement problems. 

The purpose of the research done on this contract was to extend the types 
of models available for model-based measurement. Two types of extensions were 
considered. The first was an extension of item response theory models to the 



case where the measurement device was not assumed to be measuring a single 
dimension. These models were labelled multidimensional item response theory 
(MIRT) models. 

The second type of extension was to cases where sets of related items 
were considered as a unit. These" related sets" of items were assumed to be 
measuring educational constructs that could be arranged into a hierarchy that 
facilitated learning. These models could be used to determine the 
interrelationship between the constructs in the hierarchy and the level that 
must be reached on each construct before a person should be moved on to the 
next higher level of the hierarchy. Models for tests used with hierarchically 
arranged instructional units were labelled models for hierarchically 

o 

structured tests (HST). 

The approach taken to develop and evaluate the MIRT and HST models was to 
first logically evaluate the characteristics of potential models, then to 
develop estimation procedures fot the parameter of the models, and finally to 
evaluate the models on their ability to describe real test data. These steps 
were performed separately for a wide class of models of each type. The 
results of the research will now be described for each type of model, with the 
analysis of the MIRT models being presented first. Only a summary of the 
outcome of the research will ^be presented here, but references will be made to 
papers and technical reports that contain the details of the research efforts. 

The Development and Evaluation of MIRT Models 

The class of possible multidimensional, probabilistic models of the 
interaction between a person and a test item is essentially infinite in 



3 



size. Any expression tnat maps a vector of abilities into a probability could 
be considered as a MIRT model. 

Therefore, the first step in the research effort was to limit the 
possible models to a manageable subset. This was done by reviewing the 
literature to determine what MIRT models had been proposed. The review 
identified three general classes of models that had been suggested for use 
with multidimensional data. 

The first of the classes of models considered were extensions of the 
general model proposed by Rasch (1961). This model, in its most general form, 
is given by 

where probability of response x^^ given the values of 

vector parameters 6^ and a^; 6_. is a vector of parameters that describes the 
characteristics of person j; is a vector of parameters that describes item 
Y (®j f^j) is a normalizing function defined 

Y(e..a^) = I e^*^\3^''i ' '^^a^^ (2) 

that ensures that the sum of the probabilities of the responses to this item 
Is equal to 1.0; ♦(^y) a vector of scoring weights that indicates the 
value to be given to each response to the items when considering the 
estimation of the ability parameters; ^'C^y ) a vector of scoring weights 
that indicates the value to be given to each response to the item when 
considering the estimation of Item parameters; X(x^j) is a matrix of scoring 
weights that indicates the value to be given to different products of the 
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elements of 6^ and o^; and P(^jLj) a constant that is used to set the origin 
of the linear function defined by the exponents This equation defines a very 
general class of models that specifies the dimensionality of the complete 
latent space by a linear function in the exponent of the logistic model 
form. Note that this model allows one ability to compensate for another in 
the metric of 6 ^ . That is, a high value of can compensate for a low value 
of 9^ in the linear function of 0, defined by 



The same type of linear compensation is present for the item parameters. 

V 

The second class of models considered was proposed by Mulaik (1972). 
This class of models is of the form 



m 



1 . I e^^jk" ^ik) 



where x^, = 0,1; m is the number of dimensions; and all of the other terms 
have been defined previously. This model specifies the dimensionality of the 
complete latent space as a sum of exponential terms. Ability and item 
parameters can also compensate for each other in this model, but the 
compensation occurs on an exponential scale. An interesting point to note Is 
that if each exponent is zero in this model, the probability of a correct 
response is m/(m + 1). Thus, as the number of dimensions, m, increases, the 



/ 

5 

probability of a correct response increases unless all of the^erson and item 
parameters are rescaled. For the model presented in Equation 1, the 
probaoility is always .5 when the exponent is zero* 

The third class of mojdels »that was considered was proposed by Sympson 
(1978) and in a slightly different form by Whitely (1980). This class of 
models is of the general form given by 



P(XjLj=l|Qj» ^i' ^i' ^i^ " ^i ^^""^i^ k^l — 



(^jk - ^k^ 



^k^^jk"- ^k^ 



(5) 



1 + e 



where a^ is a vector of discrimination parameters, b^ is a vector of 
difficulty parameters, is the lower asymptote of the probability function, 
and all of the other terms have been defined previously* This class of models 
determines the probability of a response based on abilities in a 
multidimensional space as the product of a series of probability like terms* 
These terms are, in effect, the probability of the response to the item if the 
' item only required the one dimension* The overall probability is the product 
of the probabilities on each dimension* tf the exponent is zero on each 
dimension, the probability will be c^ + (1 - c^) (*5)T Thus, the probability 
of a correct response will be reduced as each additional dimension is 
included, unless the parameters are rescaled for each level of dimensionality* 

Since the models given in Equations 4 and 5 both require a rescaling of 
the ability scales with each change in dimensionality, and because both of 
these models present some very difficult problems in parameter estimation, 
they were removed from initial consideration and the model presented in 
Equation 1 became the focus of research effort* 

ERIC ' , ^ 
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Analysis of the General Rasch Model 

The model presented in Equation 1 defines a very rich class of special 
cases. By selectively setting the weight functions to zero, many different 
possi*ble models can be derived, each of wMch have different properties. Each 
of these special cases was studied both through a mathematical analysis of the 
equation for each model and through a statistical analysis of simulated data 
generated using each model. The results of these analyses were reported in a 
t-ichnicel report and in a series of papers presented at professional 
meetings. The full references to the report and the papers are given below. 

McKinley, R. L. and Reckase, M. D. (1982). The use of the general. Rasch model 
with multidimensional Item response data (Research Report ONR 82-1). Iowa 
City, lA; The American College Testing Program. 

McKinley, R. L. and Reckase, M» !>• (1982, March). Multidimensional latent 
trait models. Paper presented at the meeting of the National Council on 
Measurement ih Education, New York. 

McKinley, R. L. and Reckasi, M. D. (1982, May). An analysis of the 

characteristics of a family of IRT models . Paper presented at the meeting 
of the Psychometric Society, Montreal. 

The results of these analyses showed that two special cases of the 
general Rasch were capable of modeling realistic multidimensional item 
response data. The first case uses only th(5 0' x(x, , )cr. and ♦(x.,)'a terras 
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of the general model. The weights for the other terms were set to zero. The 
model for this case is given by 



m m 



1 ^ ^ ''ik^k■*■ ^ °i m+k^ 



where the symbol s^^have bean defined earlier. This form of the model can be 
written in the more familiar form given by 



m 

I + e ( i ^ik^jk^ '^i^ 
k=l 

m 

• ^i - \ = \ ,\ m ■^ k,^ ^ ^^=1 - ^(«i'V 



where a., = a^, 
ik ik 



and a^j^ and b^j^ can be interpreted as the a- and b-parameters from 
unidimensional IRT models. Equation 7 can also be thought of as a 
multidimensional extension of the two-parameter logistic' model; therefore, it 
has been labelled the M2PL model. 

The second special case of the general Rasch model Chat was found to 
model multidimensional item response data uses only Che ^^^ij^'^j 
and terms from the general model. This model is of the form 



(8) 
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V 



where all of the terms have bee:^* defined previously. This model has been 
labelled the "cluster model" because in order for it to model multidimensional 
data, XjLj must be the response string for a cluster of items rather than the 
response to a single item. If the item cluster contains two dichotomously 
scored items, the possible x^^ responses would be 0,0; 0,1; 1,0; and 1,1. For 
each of, these r.esponses, a different weight function would be available for 

the 9- and o-vectors. 

Although the cluster model was very promising, it had one difficulty that 
made it less attractive. In order to use the model, items had to be 
clustered, and no rigorous means for doing the clustering has been 
developed. Therefore, research efforts concentrated on the M2PL model. 

Estimation of Model Parameters 

In order for a model to be useful, it must be possible to estimate the 
parametets of the model. Once the M2PL model was selected as the model for 
further research efforts, work was becrun on developing procedures for 
estimating the model parameters. Twc/dlff^erent approaches were taken to solve 
' the estimation problem: (a) unconditionai maximum likelihood, and (b) 

♦ conditional maximum likelihood. Once computer programs were developed for 

\ 

these two approaches, they were validated usli^ both simulated test data 
generated from the\M2PL model, and real test data that were selected because 
of their multivariate^ properties, the estimation procedures and -..o results 
of the program validation studies were presented in the publications and 
papers listed below. 

V 
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McKinley, R. L. and Reckase, M<^ D. (1983). MAXLOG: a computer program for 
the estimation of the parameters of a multidimensional logistic model. 
Behavior Research Methods and Instrumentation, 15(3), 389-390. 



McKinley, R. L. and Reckase, M. D. (1983). An application of a 

multidimensional extension of the two-parameter logistic latent trait model 

(Research Report ONR83-3). Iowa City, lA: The American College Testing 
Program. 



Reckase, M. D. and McKinley, R. L. (1982, July). Some latent trait theory in 
a multidimensional latent space. Paper presented at the Invitational 
Conference on IRT/CAT, Wayzata, MN. 



Reckase, M. D. and McKinley, R. L. (1982, August). The feasibility of a 

multidimensional latent trait model . Paper presented at the meeting of the 
American Psychological Association, Washington, D.C. 



McKinley, R. L. (1983, April). A multidimensional extension of the two- 
parameter logistic latent trait model . Paper presented at the meting of 
the National Council on Measurement in Education, Montreal. 



McKinley, R. L. and Reckase, M. D. (1983, April). The use of IRT analysis on 
dichotomous data from multidimensional tests . Paper presented at the 
meeting of the American Educational Research Association, Montreal. 



ERIC 



13 



10 



The results of these studies showed that both the unconditional and 
conditional maximum likelihood procedures could be used to estimate the Item 
and ability parameters of the M2PL model, but that the unconditional maximum 
likelihood procedure required somewhat less computer time. However', both 
procedures require fairly extensive computer facilities, and as the number of 
dimensions in the model increased, the computer time required became 
prohibitive. It was clear that imp>-oved estimation procedures were needed if 
the M2PL model was to be widely used. 

The validation of the estimation procedures yielded uniformly good 
results when simulated test results wece used. However, when real test data 
were analyzed, the results were inconsistent. Some studies gave readily 
interpretable results that were in many ways similar to factor analytic 
results. In other studies anomolies appeared , such as highly negatively 
correlated ability estimates that suggested that added constraints were needed 
to control the estimation process. 

In order to study the estimation process in more detail, the M2PL 
procedure was used to analyze simulated test data that had been produced using 
a multivariate ability distribution that had varying degrees of correlation 
between the abilities. The results of the study wera presented in the 
following report. 

McKinley, R. L. and Reckase, M. D. (198A). An investigation of the effect of 
correlated abilities on observed test characteristics (Research Report ONR 
8A-1). Iowa City, Iki The American College Testing Program. 
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The study showed that the dimensionality of both the items and the 
examinee population was important in interpreting the results of an M2PL 
analysis. If each item were a relatively pure measure of an ability, the 
procedure obtained good estimates of Lhe ability parameters, even when they 
were correlated. But, as the correlation between ability estimates increased, 
there was some deterioration of the accuracy of the estimates. When each item 
ipoasured more than one ability, the effect of correlated abilities was more 
extreme. As the correlation between abilities increased, the M2PL solution 
tended to collapse to a single dimension. The results seemed to imply the 
need for procedures for oblique rotations to improve the recovery of the 
ability dimensions. 

Interpretation of the Model Parameters 

When a MIRT model is used, estimates can be obtained for the ability and 
the item parameters. The ability parameter estimates can be interpreted in a 
fairly straightforward manner as the amount of ability a person has on each 
dimension. The item parameter estimates, however, do not have the same 
intuitive meaning. Therefore, a major part of this project dealt with 
determining the MIRT .model analogs to the unidimensional IRT item parameters 
and the measures of quality, such as item and test information^ The results 
of the work in this area were presented in the following documents. 

McKinley, R. L. and Reckase, M, D. (1983). An extension of the two-parameter 
logistic model to the multidimensional latent space Research Report 0NR83- 
2). Iowa City, lA: The American College Testing Program. 
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Reckase, M. D. and McKinley, R* L. (1983, April). The definition of 

difficulty and discrimination for multidimensional item response theory 
modals* Paper presented at the meeting of the American Educational 
Research Association, Montreal. 

Reckase, M. D. and McKinley, R* L. (1983, June). The item difficulty concept 
generalized to the multidimensional latent space . Paper presented^ at the 
meeting of the Psychometric Society, Los Angeles. 

Reckase, M. D. and McKinley, R. L. (1984, June). Multidimensional difficulty 
as a direction and a distance . Paper presented at the meeting of the 
Psychometric Society, Santa Barbara, CA. ^" .^^ 

Initial work in this area concentrated on deriving a direct 
generalization of the interpretations of the difficulty and discrimination 
parameters and item and test information from the unidimensional item response 
theory models to the MIRT models. Since the difficulty of an item was defined 
for th(i unidimensional models as the point on the ability scale corresponding 
to the point of inflection of the item characteristic curve, multidimensional 
difficulty was conceptually thought of as the point of inflection of the 
multiditnensional item response surface (IRS). An analysis of this approach 
quickly made two important points evident. First, for an IRT there is not a 
single point of inflection, but rather a locus of points of inflection. 
Depending upon the MIRT model and the dimensionality being considered, this 
locus of points of inflection could be a straight line, a curve, a hyperplane, 
or a hypersurf ace. The complexity of the locus of points of inflection made 
its practical application difficult. 

16 
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The second point that became evident was that the locus of points of 
inflection could change with the direction taken relative to the surface in 
the multidimensional space. This is a direct consequence of the fact that the 
slope at a^pMnt on the IRS is different in different directions. The 
direction in the space is ooe way of indicating the composite of abilities 
that is of interest. 

In order to take these two points into account, a definition of 
multidimensional difficulty was derived that was based upon a vector 
conceptualization. The multidimensional difficulty of an item was defined as 
the direction from the origin of the multidimensional space to the point of 
steepest slope and the distance from the ori^^in to the point of steepest 
slope. Discrimination of an item was related to the slope in the difficulty 
direction at the point of the steepest slope. Information was also given a 
directional interpretation. For a group centered at the origin of the space, 
an item is most informative in the^^d^Tf ficulty direction. The item information 
can also be determined in any other direction, but the maximum information 
will be less than in the direction indicated by the multidimensional 
difficulty. 

The definitions of multidimensional difficulty, discrimination, and 
information are general enough that they apply to any MIRT model that is 
monotonically increasing in probability with an increase in any ability 
dimension. The definition also includes the unidimensional definitions as 
special cases. 

Summary and Conclusions 

This portion of the research project accomplished several important tasks 
In the development of MIRT. A number of models were analyzed and the 
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multidimensional extension of the two-parameter logistic model was selected as 
a promising model for future work* Estimation procedures were developed for 
this model and the results were validated using simulated and real test 
data# A theoretical foundation was layed for an interpretation of the Item 
parameters of the flIRT models, and definitions of multidimensional item 
difficulty, discrimination, and information were developed* At this point, a 
sufficient framework has been developed to make multidimensional item response 
theory a viable technique* 

Although substantial advances have been made in the area of MIRT, even 
more work is left to be done* The current estimation programs requ»:^e 
excessive amounts of computer time when more than two or . three dimensions are 
specified for a model* Work needs to be done to make estimation of the 
parameter more efficient* Procedures are needed determine the appropriate 
number of dimensions for a set of test data, and t,*.*ocedures for Indicating the 
fit of the models to the data are needed* A related question is .whether the 
M2PL model is an accurate representation of the interaction between a person 
and an item* This model implies that one ability can compensate for 
another* Perhaps a model of this type is not appropriate* These and other 
questions will be addressed in future work* 

Models for Performance on 
Hierarchically Structured Training Materials 



Programs of instruction a^e often composed of many short, homogenous 
Instructional unitQ that have been arranged according to the logical 
Interrelationships of the content* In many cases, short tests are given to 
ilpt^^nnino oarh studonl*^ level of competence on a unit of instruction, and the 
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scoires on the tests are used to route the students through the units of 
instruction • The purpose of this component of the project was Lo evaluate an 
IRT-type model that had potential for assisting In determining the 
interrelationships between the instructional units and in determining the 
decision points that should be used with each unit test to minimize routing 
error. The model treats each unit, or module, of instruction as a complex 
item and hypothesizes a particular mathematical form for the interrelationship 
between performance on one module and the probability of successfully passing 
the next module in the instructional program. 

The first step in the evaluation of this model for performance in 
instructional programs was to review the literature in the area called 
"learning hierarchies" to determine what procedures were, currently being used 
to evaluate the interrelationships between units of instruction and to set 
passing scores on the unit tests. The information obtained from the review 
would serve as a basis for comparison for the results obtained from the 
proposed model. The review of the literature was presented in the following 
report. 

Reckase, M. D. and McKinley, R. L. (1982). T he validation of learning 
hierarchies (Research Report OMR 82-2). Iowa City, lA: The American 
College Testing Program. 

The review of the literature indicated that there were two general types - 

« 

of procedures that had been used to indicate the relationships between 
instructional units; those based on coefficients of dependence, and those 
based on a more complete description of the relationships between units of 
Instruction, usually a mathematical model. The procedures based on 
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coefficients of dependence were found to provide insufficient Information for 
validating the sequence of instructional units, or for setting passing 
scores • The procedures based on mathematical models were ^^V^^ have more 
potential, but the currently available procedures did not seem to meet the 
needs of instructional programs. There seemed to be a clear need £or a 
procedure that could be used to arrange units of instruction into a hierarchy 
based upon the prerequisite knowledge required by each unit of instruction, 
and that could be used to set passing scores for each unit that would improve 
the efficiency and accuracy of the routing process. The model proposed and 
evaluated during this research effort was designed to perform these functions. 

The Module Characteristic Curve Model 

The basic idea behind the proposed model for the interrelationship 
between modules of instruction is that if two modules form a learning 
hierarchy, performance on the higher level instructional module is dependent 
upon prerequisite knowledge obtained from the lower level module of 
instruction. Thus, if sufficient knowledge has not been gained on the lower 
level module, a high level of performance cannot be exhibited on the higher 
level modble of instruction. This implies that success on the higher module 
is related to the level of performance on the lower module. 

The probabilistic model that was hypothesized to describe the 
relationship between hierarchically related instructional modules is given by 



C. . (l-Cj 



(9) 
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where PjC^j^k^ probability of passing module j given level of 

performance 6^^^ of examinee i on prerequisite module k, Cj is the probability 
of parsing module j if the examinee has not acquired any knowledge in module 
k, ej is the probability of passing module j if the examinee has mastered 
module k, D = a^ is ^a parameter related to the strength of the 

relationship between the two modules, and bj is the difficulty of the passing 
score used on module i* This model predicts the probability that an examinee 
will pass module j based on his/her performance on module k« 

In order to use this model, estimates of achievement are first obtained 



on module k. This can either be done by analyzing the module k test using an 
IRT model, or by converting the raw scores on module k to z-scores« These 
achievement measures are then used as known values and the model parameters 
are estimated using a maximum likelihood estimation procedure • 

A very low a-parameter estimate is an indication that Che two modules are 
not very highly related. A high a-value indicates that knowledge on module k 
is very important for module j« A high estimate for the c-parameter indicates 
that examinees can perform well on module j even without mastering module k« 
A low c-value indicates that an examinee cannot perform well on module j 
unless knowledge has been acquired on module k.' 

Estimates of the e-parameter indicate the maximum probability of passing 
the j module given that the examinee has mastered module k. Low values 
indicate that module k contains only a small portion of the information needed 
to pass module High values indicate that module k includes most of the 
information needed to pass module j< 

The b-parameter estimates indicate the point on the module k scale that 
Best distinguishes between persons who pass or fail module This point will 
change with changes in the passing score on module j« The point on the module 



\ 




k scale specified by the b-parameter is the suggested decision point on module 
k for routing to module j if misclassification errors in either direction are 
considered equally serious. 

In order to evaluat^e^ this model, it was applied to both simulated and 
real test data to determine whether the estimation procedures worked properly, 
and whether it: realistically represented actual test results. The outcome of 
these studies were presented in the following documents. 

McKinley, R. L. and Reckase, M. D. (1984). A latent trait model for 

secjuentlally arranged vinits of instruction . Iowa City, lA: The American 
College Testing Program. 

McKinley, R. L. and Reckase, M. D. (1984, April). A latent t rait model for 
use with sequentially arranged units of instruction . Paper presented at 
the meeting of the American Educational Research Association, New Orleans. 

The studies showed that the parameters of the model could be accurately 
estimated and that for one set of real test data, the model gave very 
reasonable results. There was some indications, however, that the upper and 
lower asymptote parameters might not be needed. It may be possible to 
simplify the moill to a two-parameter logistic form. 

Summary and Conclusions 

A model for the relationship between modules of instruction that are 
hierarchically related was proposed and evaluated using both simulated and 
real test data. The results of the studies showed that the model parameters 
could be accurately estimated and that the mod^ was a good representation of 
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real test data that should be hierarchically related* However, the upper and 
lower asymptotes did not appear to be needed for the particular real data set 
that was analyzed. Further studies need to be done to determine whether this 
is a general finding applicable to all hierarchically arranged modules, or 
whether it only applies to this case* If the ,c- and e-parameters are not 
needed, the model can be simplified to a two-paratneter logistic model. 

One problem with the use of the model became evident with the analysis of 
the real test data. In order to accurately estimate *the parameters of the 
model, examinees must be routed to the higher level unit of Instruction even 
when they have not performed well on the lower level unit. This Is poor 
educational practice and. In many cases, this data collection procedure cannot 
be followed. This makes It difficult to obtain data for use In estimating the 
parameters of the model. It may be that the model will have to be modified to 
accomodate the routing procedures that are currently being used 1(1 modularized 
Instructional programs. 
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